Skip to content

feat: Add flag overrides to the contract test services - #530

Draft
kinyoklion wants to merge 1 commit into
rlamb/overrides-python-file-sourcefrom
rlamb/overrides-python-contract-tests
Draft

kinyoklion wants to merge 1 commit into
rlamb/overrides-python-file-sourcefrom
rlamb/overrides-python-contract-tests

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

This is the last step of the flag overrides port described by the OVERRIDE specification. It is based on the file source branch because the phases are stacked; retarget to feat/overrides once that branch merges.

Both contract test services (the Flask service for LDClient and the aiohttp service for AsyncLDClient) advertise the flag-overrides capability and accept the overrides object in the SDK configuration as the test harness service specification describes: filePaths (required, in precedence order) and the optional duplicateKeysHandling, changeDetection, and pollIntervalMs properties. The values configure a FileOverrideSourceBuilder on the FDv2 data system through ConfigBuilder.overrides(...) or AsyncDataSystemConfig.override_source. An overrides object without a dataSystem configuration is rejected with an error, because overrides are an option of the FDv2 data system only.

Verification with the prebuilt harness from sdk-test-harness branch rlamb/sdk-2657/flag-overrides-tests (v3.2.0-alpha.10):

  • Sync service: -run 'flag overrides' 38 ran, all passed; full suite 4809 total, 15 skipped, 4794 ran, all passed.
  • Async service: -run 'flag overrides' 38 ran, all passed; full suite 4807 total, 16 skipped, 4791 ran, all passed.

The 15 tests skipped by the overrides runs are the other top-level suites excluded by the filter. The tests skipped in the full runs are those for capabilities the services do not advertise (strongly-typed, tls:verify-peer, service-endpoints, user-type, track-hooks, wrapper, and on the async service hook-environment-id), the long-running tests, and the persistent data store tests, all of which were skipped before this change too.

SDK-3250


Note

Overview
Both contract-test services (sync Flask and async aiohttp) now advertise the flag-overrides capability and map harness overrides config into the FDv2 data system.

When dataSystem is present, an optional overrides block builds a FileOverrideSourceBuilder from filePaths and optional duplicateKeysHandling, changeDetection, and pollIntervalMs—wired via datasystem.overrides(...) on the sync client and AsyncDataSystemConfig.override_source on the async client. overrides without dataSystem raises ValueError, since overrides are FDv2-only.

No production SDK behavior changes; this is harness wiring so sdk-test-harness flag-overrides suites can run against Python.

Reviewed by Cursor Bugbot for commit dac6abe. Bugbot is set up for automated code reviews on this repo. Configure here.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 021778c to 2074b74 Compare September 28, 2026 20:29
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from b5ff7c2 to 0a9c386 Compare September 28, 2026 20:29
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 2074b74 to 1950d45 Compare September 30, 2026 20:31
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 0a9c386 to bad30c5 Compare September 30, 2026 20:31
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 1950d45 to 3e47045 Compare October 1, 2026 23:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from bad30c5 to a355c3d Compare October 1, 2026 23:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 3e47045 to cfb781a Compare October 3, 2026 01:18
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from a355c3d to dac6abe Compare October 3, 2026 01:19
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit dac6abe. Configure here.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from cfb781a to ea4dc19 Compare October 13, 2026 21:02
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from dac6abe to 3de3ec9 Compare October 13, 2026 21:02
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from ea4dc19 to 3de11ab Compare October 13, 2026 22:40
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch 2 times, most recently from 8d2c06c to 6190743 Compare October 15, 2026 22:25
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from bb5a108 to 5e42ebc Compare October 16, 2026 21:34
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 6190743 to ac6bdd0 Compare October 16, 2026 21:34
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 5e42ebc to 314dea2 Compare October 28, 2026 22:17
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from ac6bdd0 to 0d759ba Compare October 28, 2026 22:17
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 314dea2 to 4bace41 Compare October 28, 2026 22:40
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch 2 times, most recently from 1dbbf2c to 1f4ce1a Compare October 28, 2026 22:53
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 99b851f to fac9ec8 Compare October 29, 2026 16:48
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from 1f4ce1a to d1b617b Compare October 29, 2026 16:48
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from fac9ec8 to 0415ec1 Compare October 29, 2026 17:16
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-contract-tests branch from d1b617b to 0d55468 Compare October 29, 2026 17:16
kinyoklion added a commit to launchdarkly/sdk-test-harness that referenced this pull request Nov 3, 2026
…thout the init key (#539)

<!-- TERMAG_PR_BODY_BEGIN -->
## Summary

Overrides only add to what the SDK reports: a flag that the override
layer does not hold evaluates as it does without an override source, and
the all-flags state holds every flag it holds without one, plus the
overridden flags. A client that has not initialized is where SDKs broke
this, and only when the store holds flags without being initialized, for
example a persistent store that holds data but no init key. The flag
overrides suite uses the SDK's in-memory store, which is empty until a
data source provides data, so it could not tell a read of the override
layer apart from a read of everything the store returns.

A new `flag overrides` group in the persistent data store suite
(`server_side_persistence_overrides.go`) runs when the test service has
the `flag-overrides` capability. It writes flags to the store without
the init key and gives the client a data source that never provides
data. Each test compares a client with an override source to a client
without one against the same store, so it holds whatever an SDK reports
for an uninitialized store:

- **a flag without an override evaluates as it does without an override
source**: an override file that holds a different flag leaves the
evaluation of a store flag unchanged (OVERRIDE Requirements 1.5.1 and
1.5.3).
- **an empty override source leaves the all-flags state unchanged**
(Requirement 1.2.2).
- **overrides add to the all-flags state and make it valid**: the state
holds exactly the flags of the client without a source, with override
values for the overridden keys, plus the flags that exist only in the
override layer, and it is valid (launchdarkly/sdk-specs#280,
Requirements 1.4.1.2 and 1.4.1.3).

The flag overrides suite gains **evaluate all flags returns only the
override flags as a valid state** in its uninitialized client group,
where the store is empty and every SDK reports only the override flags.

## Results

Run with `-enable-persistence-tests` against Redis.

| Test service | flag without an override | empty override source |
overrides add |
|---|---|---|---|
| Go `v7` | fails: served from the store | fails: valid state of the
store's flags | fails: the store's flags join the overrides |
| Go launchdarkly/go-server-sdk#491 | passes | passes | passes |
| Node `main` | passes | fails: valid state of the store's flags |
passes |
| Node launchdarkly/js-core#2132 first commit (`57498a9e`) | passes |
passes | fails: drops a store flag the state holds without overrides |
| Node launchdarkly/js-core#2132 | passes | passes | passes |
| Python (top of the launchdarkly/python-server-sdk#530 stack, sync and
async) | passes | passes | passes |
| Ruby (top of the launchdarkly/ruby-server-sdk#454 stack) | passes |
passes | passes |

The Java (launchdarkly/java-core#225) and .NET (`dotnet-server-sdk-v9`)
test services do not support persistent stores, so the group skips
there; the new uninitialized client test passes on both. The full Redis
persistent data store suite and the flag overrides suite pass against Go
#491 at `30d4f216` (131 tests run). `golangci-lint` reports no issues.

Go and Node fail these tests until launchdarkly/go-server-sdk#491 and
launchdarkly/js-core#2132 are merged, for test services whose CI runs
the persistent data store suite with this harness version.

[SDK-3419](https://launchdarkly.atlassian.net/browse/SDK-3419)
<!-- TERMAG_PR_BODY_END -->

<!-- termag: box=i-0575d1c54853058d7 project=sdk-test-harness -->
<sub>Opened from termag</sub>


[SDK-3419]:
https://launchdarkly.atlassian.net/browse/SDK-3419?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Adds harness coverage for **file flag overrides** when the SDK never
initializes but a **persistent store already holds feature data without
the `$inited` key**—a case the existing in-memory override suite could
not distinguish from “read everything from the store.”
> 
> A new **`flag overrides`** group under the server-side persistent data
store suite (`server_side_persistence_overrides.go`, gated on
`flag-overrides`) seeds Redis/Consul/Dynamo with flags, uses a blocking
unavailable data source, and compares clients with and without override
files via `uninitializedClient`. Scenarios assert: store flags not in
the override file behave like no override source; an empty override file
leaves **evaluate all flags** unchanged; overrides merge into all-flags
state, mark it **valid**, and apply override values while retaining
other store flags.
> 
> The standalone flag-overrides **`uninitialized client`** group gains
**evaluate all flags returns only the override flags as a valid state**
(empty store, overrides-only valid state). The persistent suite
registers the new group from `server_side_persistence_base.go`.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
09a8e02. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Both contract test services advertise the flag-overrides capability and
accept the overrides object in the SDK configuration: filePaths, and the
optional duplicateKeysHandling, changeDetection, and pollIntervalMs
properties, which configure the file-based override source on the FDv2 data
system. An overrides object without a data system configuration is rejected,
because overrides are an option of the FDv2 data system only.
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-python-file-source branch from 0415ec1 to a8e9294 Compare November 3, 2026 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant